ARROW-90: [C++] Check for SIMD instruction set support #50

xhochy · 2016-03-29T13:35:16Z

This also adds an option to disable the usage of a specific instruction
set, e.g. you compile on a machine that supports SSE3 but you want to
use the binary also on machines without SSE3. (Distribution packagers
will love that option!)

This also adds an option to disable the usage of a specific instruction set, e.g. you compile on a machine that supports SSE3 but you want to use the binary also on machines without SSE3.

wesm · 2016-03-30T21:19:50Z

cpp/CMakeLists.txt

+endif()
+if (CXX_SUPPORTS_ALTIVEC AND ARROW_ALTIVEC)
+ set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} -maltivec")
+endif()


We're probably also going to want some defines so that we can include or exclude code that depends on these features, but doesn't need to be in this patch

What implications does ARROW_SSE3=on by default have if we also add ARROW_SSE42 for SSE 4.2? I figure that -msse4.2 -msse3 is equivalent to just -msse3 but let me know if that is not the case

-msse4.2 -msse3 can be used together and they will enable SSE3 and SSE4.2 support. The semantics of those switches are "also use this instruction set". It may even be that -msse4.2 implies -msse3 so that -msse4.2 -msse3 is actually equivalent to -msse4.2. In general those -m* options add up together to enable more instructions sets, not less.

thank you, good to know

wesm · 2016-04-01T00:48:27Z

+1

closes apache#43 and closes apache#50 This PR also implements 1) PARQUET-532: Null values detection needs to be fixed and tested 2) PARQUET-502: Scanner segfaults when its batch size is smaller than the number of rows 3) PARQUET-526: Add more complete unit test coverage for column Scanner implementations 4) PARQUET-531: Can't read past first page in a column Author: Deepak Majeti <deepak.majeti@hp.com> Closes apache#62 from majetideepak/PARQUET-538 and squashes the following commits: 1e56f83 [Deepak Majeti] Trigger notification 6478a7c [Deepak Majeti] TYPED_TEST 1d14171 [Deepak Majeti] Added Boolean Test and Scanner:Next API d1da031 [Deepak Majeti] lint issue 45f10aa [Deepak Majeti] Reproducer for PARQUET-502 88e27c6 [Deepak Majeti] formatting 8aac435 [Deepak Majeti] PARQUET-526 dca7e2d [Deepak Majeti] PARQUET-532 and PARQUET-502 Fix a622021 [Deepak Majeti] Reverted PARQUET-524 and addressed comments 859c1df [Deepak Majeti] minor comment edits d938a13 [Deepak Majeti] PARQUET-538 df1fbd7 [Deepak Majeti] Templated single page tests 8548e3c [Deepak Majeti] PARQUET-524 c265fea [Deepak Majeti] fixed PARQUET-499 bugs

closes apache#43 and closes apache#50 This PR also implements 1) PARQUET-532: Null values detection needs to be fixed and tested 2) PARQUET-502: Scanner segfaults when its batch size is smaller than the number of rows 3) PARQUET-526: Add more complete unit test coverage for column Scanner implementations 4) PARQUET-531: Can't read past first page in a column Author: Deepak Majeti <deepak.majeti@hp.com> Closes apache#62 from majetideepak/PARQUET-538 and squashes the following commits: 1e56f83 [Deepak Majeti] Trigger notification 6478a7c [Deepak Majeti] TYPED_TEST 1d14171 [Deepak Majeti] Added Boolean Test and Scanner:Next API d1da031 [Deepak Majeti] lint issue 45f10aa [Deepak Majeti] Reproducer for PARQUET-502 88e27c6 [Deepak Majeti] formatting 8aac435 [Deepak Majeti] PARQUET-526 dca7e2d [Deepak Majeti] PARQUET-532 and PARQUET-502 Fix a622021 [Deepak Majeti] Reverted PARQUET-524 and addressed comments 859c1df [Deepak Majeti] minor comment edits d938a13 [Deepak Majeti] PARQUET-538 df1fbd7 [Deepak Majeti] Templated single page tests 8548e3c [Deepak Majeti] PARQUET-524 c265fea [Deepak Majeti] fixed PARQUET-499 bugs Change-Id: I72ba1fb655e851fc6045d925ac5715f5b535ad7f

This PR enables tests for `ARROW_COMPUTE`, `ARROW_DATASET`, `ARROW_FILESYSTEM`, `ARROW_HDFS`, `ARROW_ORC`, and `ARROW_IPC` (default on). #7131 enabled a minimal set of tests as a starting point. I confirmed that these tests pass locally with the current master. In the current TravisCI environment, we cannot see this result due to a lot of error messages in `arrow-utility-test`. ``` $ git log | head -1 commit ed5f534 % ctest ... Start 1: arrow-array-test 1/51 Test #1: arrow-array-test ..................... Passed 4.62 sec Start 2: arrow-buffer-test 2/51 Test #2: arrow-buffer-test .................... Passed 0.14 sec Start 3: arrow-extension-type-test 3/51 Test #3: arrow-extension-type-test ............ Passed 0.12 sec Start 4: arrow-misc-test 4/51 Test #4: arrow-misc-test ...................... Passed 0.14 sec Start 5: arrow-public-api-test 5/51 Test #5: arrow-public-api-test ................ Passed 0.12 sec Start 6: arrow-scalar-test 6/51 Test #6: arrow-scalar-test .................... Passed 0.13 sec Start 7: arrow-type-test 7/51 Test #7: arrow-type-test ...................... Passed 0.14 sec Start 8: arrow-table-test 8/51 Test #8: arrow-table-test ..................... Passed 0.13 sec Start 9: arrow-tensor-test 9/51 Test #9: arrow-tensor-test .................... Passed 0.13 sec Start 10: arrow-sparse-tensor-test 10/51 Test #10: arrow-sparse-tensor-test ............. Passed 0.16 sec Start 11: arrow-stl-test 11/51 Test #11: arrow-stl-test ....................... Passed 0.12 sec Start 12: arrow-concatenate-test 12/51 Test #12: arrow-concatenate-test ............... Passed 0.53 sec Start 13: arrow-diff-test 13/51 Test #13: arrow-diff-test ...................... Passed 1.45 sec Start 14: arrow-c-bridge-test 14/51 Test #14: arrow-c-bridge-test .................. Passed 0.18 sec Start 15: arrow-io-buffered-test 15/51 Test #15: arrow-io-buffered-test ............... Passed 0.20 sec Start 16: arrow-io-compressed-test 16/51 Test #16: arrow-io-compressed-test ............. Passed 3.48 sec Start 17: arrow-io-file-test 17/51 Test #17: arrow-io-file-test ................... Passed 0.74 sec Start 18: arrow-io-hdfs-test 18/51 Test #18: arrow-io-hdfs-test ................... Passed 0.12 sec Start 19: arrow-io-memory-test 19/51 Test #19: arrow-io-memory-test ................. Passed 2.77 sec Start 20: arrow-utility-test 20/51 Test #20: arrow-utility-test ...................***Failed 5.65 sec Start 21: arrow-threading-utility-test 21/51 Test #21: arrow-threading-utility-test ......... Passed 1.34 sec Start 22: arrow-compute-compute-test 22/51 Test #22: arrow-compute-compute-test ........... Passed 0.13 sec Start 23: arrow-compute-boolean-test 23/51 Test #23: arrow-compute-boolean-test ........... Passed 0.15 sec Start 24: arrow-compute-cast-test 24/51 Test #24: arrow-compute-cast-test .............. Passed 0.22 sec Start 25: arrow-compute-hash-test 25/51 Test #25: arrow-compute-hash-test .............. Passed 2.61 sec Start 26: arrow-compute-isin-test 26/51 Test #26: arrow-compute-isin-test .............. Passed 0.81 sec Start 27: arrow-compute-match-test 27/51 Test #27: arrow-compute-match-test ............. Passed 0.40 sec Start 28: arrow-compute-sort-to-indices-test 28/51 Test #28: arrow-compute-sort-to-indices-test ... Passed 3.33 sec Start 29: arrow-compute-nth-to-indices-test 29/51 Test #29: arrow-compute-nth-to-indices-test .... Passed 1.51 sec Start 30: arrow-compute-util-internal-test 30/51 Test #30: arrow-compute-util-internal-test ..... Passed 0.13 sec Start 31: arrow-compute-add-test 31/51 Test #31: arrow-compute-add-test ............... Passed 0.12 sec Start 32: arrow-compute-aggregate-test 32/51 Test #32: arrow-compute-aggregate-test ......... Passed 14.70 sec Start 33: arrow-compute-compare-test 33/51 Test #33: arrow-compute-compare-test ........... Passed 7.96 sec Start 34: arrow-compute-take-test 34/51 Test #34: arrow-compute-take-test .............. Passed 4.80 sec Start 35: arrow-compute-filter-test 35/51 Test #35: arrow-compute-filter-test ............ Passed 8.23 sec Start 36: arrow-dataset-dataset-test 36/51 Test #36: arrow-dataset-dataset-test ........... Passed 0.25 sec Start 37: arrow-dataset-discovery-test 37/51 Test #37: arrow-dataset-discovery-test ......... Passed 0.13 sec Start 38: arrow-dataset-file-ipc-test 38/51 Test #38: arrow-dataset-file-ipc-test .......... Passed 0.21 sec Start 39: arrow-dataset-file-test 39/51 Test #39: arrow-dataset-file-test .............. Passed 0.12 sec Start 40: arrow-dataset-filter-test 40/51 Test #40: arrow-dataset-filter-test ............ Passed 0.16 sec Start 41: arrow-dataset-partition-test 41/51 Test #41: arrow-dataset-partition-test ......... Passed 0.13 sec Start 42: arrow-dataset-scanner-test 42/51 Test #42: arrow-dataset-scanner-test ........... Passed 0.20 sec Start 43: arrow-filesystem-test 43/51 Test #43: arrow-filesystem-test ................ Passed 1.62 sec Start 44: arrow-hdfs-test 44/51 Test #44: arrow-hdfs-test ...................... Passed 0.13 sec Start 45: arrow-feather-test 45/51 Test #45: arrow-feather-test ................... Passed 0.91 sec Start 46: arrow-ipc-read-write-test 46/51 Test #46: arrow-ipc-read-write-test ............ Passed 5.77 sec Start 47: arrow-ipc-json-simple-test 47/51 Test #47: arrow-ipc-json-simple-test ........... Passed 0.16 sec Start 48: arrow-ipc-json-test 48/51 Test #48: arrow-ipc-json-test .................. Passed 0.27 sec Start 49: arrow-json-integration-test 49/51 Test #49: arrow-json-integration-test .......... Passed 0.13 sec Start 50: arrow-json-test 50/51 Test #50: arrow-json-test ...................... Passed 0.26 sec Start 51: arrow-orc-adapter-test 51/51 Test #51: arrow-orc-adapter-test ............... Passed 1.92 sec 98% tests passed, 1 tests failed out of 51 Label Time Summary: arrow-tests = 27.38 sec (27 tests) arrow_compute = 45.11 sec (14 tests) arrow_dataset = 1.21 sec (7 tests) arrow_ipc = 6.20 sec (3 tests) unittest = 79.91 sec (51 tests) Total Test time (real) = 79.99 sec The following tests FAILED: 20 - arrow-utility-test (Failed) Errors while running CTest ``` Closes #7142 from kiszk/ARROW-8754 Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Mege native sql engine

* Correct a typo * Fix issues in getting result and cover some corner cases * Return null if given field has no data in json string * Move out the setting for out_valid * Add missing args and re-order them * Change arg order * Fix incorrect number of args issue

This reduces the running time from about 60s to about 10s. Before: ``` 86/86 Test #50: arrow-gcsfs-test .......................... Passed 60.00 sec ``` After: ``` 71/86 Test #50: arrow-gcsfs-test .......................... Passed 9.62 sec ``` Closes #11933 from coryan/ARROW-15056-gcsfs-speedup-tests Authored-by: Carlos O'Ryan <coryan@google.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

Before: ``` 80/86 Test #50: arrow-gcsfs-test .......................... Passed 13.52 sec ``` After: ``` 64/86 Test #50: arrow-gcsfs-test .......................... Passed 4.95 sec ``` Closes #11959 from coryan/ARROW-15113-gcsfs-speedup-tests-take2 Authored-by: Carlos O'Ryan <coryan@google.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

Before: ``` 80/86 Test apache#50: arrow-gcsfs-test .......................... Passed 13.52 sec ``` After: ``` 64/86 Test apache#50: arrow-gcsfs-test .......................... Passed 4.95 sec ``` Closes apache#11959 from coryan/ARROW-15113-gcsfs-speedup-tests-take2 Authored-by: Carlos O'Ryan <coryan@google.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

* Correct a typo * Fix issues in getting result and cover some corner cases * Return null if given field has no data in json string * Move out the setting for out_valid * Add missing args and re-order them * Change arg order * Fix incorrect number of args issue

ARROW-90: Check for SIMD instruction set support

6fd80d3

This also adds an option to disable the usage of a specific instruction set, e.g. you compile on a machine that supports SSE3 but you want to use the binary also on machines without SSE3.

wesm reviewed Mar 30, 2016
View reviewed changes

asfgit closed this in 79fddd1 Apr 1, 2016

xhochy deleted the arrow-90 branch March 7, 2017 16:16

jikunshang pushed a commit to jikunshang/arrow that referenced this pull request May 29, 2020

Merge pull request apache#50 from Intel-bigdata/native-sql-engine-clean

fe10d2a

Mege native sql engine

paleolimbot mentioned this pull request Jan 28, 2023

[R] Crash on MacOS (x86) when running tests with homebrew apache-arrow also installed #33903

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-90: [C++] Check for SIMD instruction set support #50

ARROW-90: [C++] Check for SIMD instruction set support #50

xhochy commented Mar 29, 2016

wesm Mar 30, 2016

wesm Mar 30, 2016

xhochy Mar 31, 2016

wesm Apr 1, 2016

wesm commented Apr 1, 2016

ARROW-90: [C++] Check for SIMD instruction set support #50

ARROW-90: [C++] Check for SIMD instruction set support #50

Conversation

xhochy commented Mar 29, 2016

wesm Mar 30, 2016

Choose a reason for hiding this comment

wesm Mar 30, 2016

Choose a reason for hiding this comment

xhochy Mar 31, 2016

Choose a reason for hiding this comment

wesm Apr 1, 2016

Choose a reason for hiding this comment

wesm commented Apr 1, 2016